{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cb969c8e-9471-442a-90d7-ca6ec2f795bc",
   "metadata": {},
   "source": [
    "# 任务\n",
    "\n",
    "1. 查看数据情况，如是否有缺失值和异常值，并对其进行处理。\n",
    "2. 将处理完的数据集输出为一个csv文件"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cfaff35-b7b3-4f2f-aad5-feb494ebeff5",
   "metadata": {},
   "source": [
    "### 1.导入需要的包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "c9ca34a2-5011-4493-967e-407163b3115e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "908aa3d7-ba58-4b2b-a76d-e1476286cf9f",
   "metadata": {},
   "source": [
    "### 2.读入数据集\n",
    "1. 从 `train.csv` 中读取训练集\n",
    "2. 从 `test.csv` 中读取测试集集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "6dbb8f11-c017-4005-ba57-01153874548f",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = pd.read_csv('train.csv')\n",
    "test_data = pd.read_csv('test.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1eac67f2-efd6-4ca8-8a05-53e3c1b10de9",
   "metadata": {},
   "source": [
    "### 3.处理缺省值并将非数字类型的属性编码\n",
    "1. PassengerId具有唯一性，因此没有预测价值，可以剔除\n",
    "2. Name每个值唯一或接近唯一，几乎没有预测价值，因此也可以剔除\n",
    "3. 剩余的属性进行缺省值处理\n",
    "4. 对非数字类型的属性编码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "ae752dee-f21e-4c74-9c7c-04a598ada0fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将 PassengerId 和 Name 从 train_data 和 test_data 中剔除\n",
    "deleted_cols = ['PassengerId', 'Name']\n",
    "train_data.drop(columns=deleted_cols, inplace=True)\n",
    "test_data.drop(columns=deleted_cols, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "b736ba34-2c32-41f4-a320-61412b533578",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将服务费用类的属性填补为均值\n",
    "numeric_cols = ['RoomService', 'FoodCourt', 'ShoppingMall', 'Spa', 'VRDeck']\n",
    "for numeric_col in numeric_cols:\n",
    "    train_data[numeric_col].fillna(train_data[numeric_col].mean(), inplace=True)\n",
    "    test_data[numeric_col].fillna(train_data[numeric_col].mean(), inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "2de6a18a-b597-4ebb-a648-20647ec708e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将年龄填补为中位数 （中位数适用于偏态分布）\n",
    "train_data['Age'].fillna(train_data['Age'].median(), inplace=True)\n",
    "test_data['Age'].fillna(train_data['Age'].median(), inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "c9c29523-3c12-4101-a6d5-75d026be2d3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将布尔值类型的属性映射为 0 和 1，并且将缺省值填补为 0\n",
    "# 对训练集进行操作\n",
    "train_bool_cols = ['CryoSleep', 'VIP', 'Transported']\n",
    "for col in train_bool_cols:\n",
    "    train_data[col].fillna(False, inplace=True)\n",
    "# 对测试集合进行操作，测试集中没有 Transported 列\n",
    "test_bool_cols = ['CryoSleep', 'VIP']\n",
    "for col in test_bool_cols:\n",
    "    test_data[col].fillna(False, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "7a086bdd-f9e6-430a-874d-aff299f7e3d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将分类列的缺省值填补为众数，再编码为 one-hot-code\n",
    "categorical_cols = ['HomePlanet', 'Destination']\n",
    "# 填补缺省值为众数\n",
    "for categorical_col in categorical_cols:\n",
    "    train_data[categorical_col] = train_data[categorical_col].fillna(train_data[categorical_col].mode()[0])\n",
    "    test_data[categorical_col] = test_data[categorical_col].fillna(test_data[categorical_col].mode()[0])\n",
    "# 编码为 one-hot-code\n",
    "train_data = pd.get_dummies(train_data, columns=categorical_cols)\n",
    "test_data = pd.get_dummies(test_data, columns=categorical_cols)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "12562764-5ac1-44c6-8b58-fd79ef766b24",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将carbin拆分成 Deck, Room, Side 三个列，再分别量化\n",
    "train_data[['Deck', 'Room', 'Side']] = train_data['Cabin'].str.split('/', expand=True)\n",
    "test_data[['Deck', 'Room', 'Side']] = test_data['Cabin'].str.split('/', expand=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "f6bdbf43-2233-4c79-889c-dffe66a46fbc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将 Deck 的缺省值填补为众数，量化 Deck\n",
    "label_encoder = LabelEncoder()\n",
    "# 先将缺省值填补为众数\n",
    "train_data['Deck'].fillna(train_data['Deck'].mode()[0], inplace=True)\n",
    "test_data['Deck'].fillna(test_data['Deck'].mode()[0], inplace=True)\n",
    "# 通过 label_encoder 量化 Deck\n",
    "train_data['Deck'] = label_encoder.fit_transform(train_data['Deck'])\n",
    "test_data['Deck'] = label_encoder.fit_transform(test_data['Deck'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "26546322-fe47-4579-aff1-ec88adfdca73",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 量化 Room，用众数补全缺省值\n",
    "train_data['Room'] = pd.to_numeric(train_data['Room'], errors='coerce').fillna(train_data['Room'].mode()[0])\n",
    "train_data['Room'] = train_data['Room'].astype(int)\n",
    "test_data['Room'] = pd.to_numeric(test_data['Room'], errors='coerce').fillna(test_data['Room'].mode()[0])\n",
    "test_data['Room'] = test_data['Room'].astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "503523c3-b053-4f3b-92c7-9271880aba5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 量化 Side，将缺省值填充为 0\n",
    "train_data['Side'] = train_data['Side'].map({'S':0, 'P': 1}).fillna(0).astype(bool)\n",
    "test_data['Side'] = test_data['Side'].map({'S':0, 'P': 1}).fillna(0).astype(bool)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "7964e8af-2ed4-4dd7-b303-902c7a2c6e87",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 删除原始属性 carbin\n",
    "train_data.drop(columns=['Cabin'], inplace=True)\n",
    "test_data.drop(columns=['Cabin'], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "b9a849d8-b534-4ef7-97e3-3a1ac79463d2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      CryoSleep   Age    VIP  RoomService  FoodCourt  ShoppingMall     Spa  \\\n",
      "0         False  39.0  False          0.0        0.0           0.0     0.0   \n",
      "1         False  24.0  False        109.0        9.0          25.0   549.0   \n",
      "2         False  58.0   True         43.0     3576.0           0.0  6715.0   \n",
      "3         False  33.0  False          0.0     1283.0         371.0  3329.0   \n",
      "4         False  16.0  False        303.0       70.0         151.0   565.0   \n",
      "...         ...   ...    ...          ...        ...           ...     ...   \n",
      "8688      False  41.0   True          0.0     6819.0           0.0  1643.0   \n",
      "8689       True  18.0  False          0.0        0.0           0.0     0.0   \n",
      "8690      False  26.0  False          0.0        0.0        1872.0     1.0   \n",
      "8691      False  32.0  False          0.0     1049.0           0.0   353.0   \n",
      "8692      False  44.0  False        126.0     4688.0           0.0     0.0   \n",
      "\n",
      "      VRDeck  Transported  HomePlanet_Earth  HomePlanet_Europa  \\\n",
      "0        0.0        False             False               True   \n",
      "1       44.0         True              True              False   \n",
      "2       49.0        False             False               True   \n",
      "3      193.0        False             False               True   \n",
      "4        2.0         True              True              False   \n",
      "...      ...          ...               ...                ...   \n",
      "8688    74.0        False             False               True   \n",
      "8689     0.0        False              True              False   \n",
      "8690     0.0         True              True              False   \n",
      "8691  3235.0        False             False               True   \n",
      "8692    12.0         True             False               True   \n",
      "\n",
      "      HomePlanet_Mars  Destination_55 Cancri e  Destination_PSO J318.5-22  \\\n",
      "0               False                    False                      False   \n",
      "1               False                    False                      False   \n",
      "2               False                    False                      False   \n",
      "3               False                    False                      False   \n",
      "4               False                    False                      False   \n",
      "...               ...                      ...                        ...   \n",
      "8688            False                     True                      False   \n",
      "8689            False                    False                       True   \n",
      "8690            False                    False                      False   \n",
      "8691            False                     True                      False   \n",
      "8692            False                    False                      False   \n",
      "\n",
      "      Destination_TRAPPIST-1e  Deck  Room   Side  \n",
      "0                        True     1     0   True  \n",
      "1                        True     5     0  False  \n",
      "2                        True     0     0  False  \n",
      "3                        True     0     0  False  \n",
      "4                        True     5     1  False  \n",
      "...                       ...   ...   ...    ...  \n",
      "8688                    False     0    98   True  \n",
      "8689                    False     6  1499  False  \n",
      "8690                     True     6  1500  False  \n",
      "8691                    False     4   608  False  \n",
      "8692                     True     4   608  False  \n",
      "\n",
      "[8693 rows x 18 columns]\n"
     ]
    }
   ],
   "source": [
    "print(train_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bbd6c19-78c0-494c-b376-ac55edbc725f",
   "metadata": {},
   "source": [
    "### 4.处理异常值\n",
    "1. 使用箱线图法判断数字类型的属性是否有异常值\n",
    "2. 将异常值替换为边界值"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6730968-7e9d-462a-bd8a-9d3078bb6472",
   "metadata": {},
   "source": [
    "#### 检查 Age 属性是否有异常值"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ded16077-ffc7-4f96-b079-bf9e199e2cda",
   "metadata": {},
   "source": [
    "1. 检查训练集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "f598ea1c-5fb7-4643-b460-12e6b7167f28",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 计算上下限\n",
    "train_Age_Q1 = train_data['Age'].quantile(0.25)\n",
    "train_Age_Q3 = train_data['Age'].quantile(0.75)\n",
    "train_Age_IQR = train_Age_Q1 - train_Age_Q3\n",
    "train_Age_lower_bound = train_Age_Q1 - 1.5 * train_Age_IQR\n",
    "train_Age_upper_bound = train_Age_Q3 + 1.5 * train_Age_IQR\n",
    "# 将检测到的异常值替换为边界值\n",
    "train_data['Age'] = np.clip(train_data['Age'], train_Age_lower_bound, train_Age_upper_bound)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91efbdbb-95a4-42c0-ae4d-3f238f1da24f",
   "metadata": {},
   "source": [
    "2. 检查测试集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "e8e5ccc9-bded-472a-9842-b0387ae5bc38",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_Age_Q1 = test_data['Age'].quantile(0.25)\n",
    "test_Age_Q3 = test_data['Age'].quantile(0.75)\n",
    "test_Age_IQR = test_Age_Q1 - test_Age_Q3\n",
    "test_Age_lower_bound = test_Age_Q1 - 1.5 * test_Age_IQR\n",
    "test_Age_upper_bound = test_Age_Q3 + 1.5 * test_Age_IQR\n",
    "# 将检测到的异常值替换为边界值\n",
    "test_data['Age'] = np.clip(test_data['Age'], test_Age_lower_bound, test_Age_upper_bound)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56bcb04a-baa0-4de6-a073-8acfc4b722d0",
   "metadata": {},
   "source": [
    "#### 检查 RoomService 属性是否有异常值"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b09be83c-ec4e-4d2b-b75a-de4d79e26959",
   "metadata": {},
   "source": [
    "1. 检查训练集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "3fdc8635-77d5-489e-9cc7-9376ac87154f",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_RoomService_Q1 = train_data['RoomService'].quantile(0.25)\n",
    "train_RoomService_Q3 = train_data['RoomService'].quantile(0.75)\n",
    "train_RoomService_IQR = train_RoomService_Q1 - train_RoomService_Q3\n",
    "train_RoomService_lower_bound = train_RoomService_Q1 - 1.5 * train_RoomService_IQR\n",
    "train_RoomService_upper_bound = train_RoomService_Q3 + 1.5 * train_RoomService_IQR\n",
    "# 将检测到的异常值替换为边界值\n",
    "train_data['RoomService'] = np.clip(train_data['RoomService'], train_RoomService_lower_bound, train_RoomService_upper_bound)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdf3ea67-bf1e-40fb-a8c6-19b0d54e791b",
   "metadata": {},
   "source": [
    "2. 检查测试集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "6f53a256-579b-4740-ae4e-ca372a2b362f",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_RoomService_Q1 = test_data['RoomService'].quantile(0.25)\n",
    "test_RoomService_Q3 = test_data['RoomService'].quantile(0.75)\n",
    "test_RoomService_IQR = test_RoomService_Q1 - test_RoomService_Q3\n",
    "test_RoomService_lower_bound = test_RoomService_Q1 - 1.5 * test_RoomService_IQR\n",
    "test_RoomService_upper_bound = test_RoomService_Q3 + 1.5 * test_RoomService_IQR\n",
    "# 将检测到的异常值替换为边界值\n",
    "test_data['RoomService'] = np.clip(test_data['RoomService'], test_RoomService_lower_bound, test_RoomService_upper_bound)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c1735b1-15ce-4ba9-b60d-8018f805318b",
   "metadata": {},
   "source": [
    "#### 检查 FoodCourt 属性是否有异常值"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a64d0d7-34f0-4d00-910a-27f3b14e3f78",
   "metadata": {},
   "source": [
    "1. 检查训练集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "e0a9e95a-44c3-4c2d-a09d-60f2dd072220",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_FoodCourt_Q1 = train_data['FoodCourt'].quantile(0.25)\n",
    "train_FoodCourt_Q3 = train_data['FoodCourt'].quantile(0.75)\n",
    "train_FoodCourt_IQR = train_FoodCourt_Q1 - train_FoodCourt_Q3\n",
    "train_FoodCourt_lower_bound = train_FoodCourt_Q1 - 1.5 * train_FoodCourt_IQR\n",
    "train_FoodCourt_upper_bound = train_FoodCourt_Q3 + 1.5 * train_FoodCourt_IQR\n",
    "# 将检测到的异常值替换为边界值\n",
    "train_data['FoodCourt'] = np.clip(train_data['FoodCourt'], train_FoodCourt_lower_bound, train_FoodCourt_upper_bound)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e41b312-34ed-4ae2-9c0a-a884385d2225",
   "metadata": {},
   "source": [
    "2. 检查测试集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "9af53427-908b-4258-9416-887fc0357fa0",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_FoodCourt_Q1 = test_data['FoodCourt'].quantile(0.25)\n",
    "test_FoodCourt_Q3 = test_data['FoodCourt'].quantile(0.75)\n",
    "test_FoodCourt_IQR = test_FoodCourt_Q1 - test_FoodCourt_Q3\n",
    "test_FoodCourt_lower_bound = test_FoodCourt_Q1 - 1.5 * test_FoodCourt_IQR\n",
    "test_FoodCourt_upper_bound = test_FoodCourt_Q3 + 1.5 * test_FoodCourt_IQR\n",
    "# 将检测到的异常值替换为边界值\n",
    "test_data['FoodCourt'] = np.clip(test_data['FoodCourt'], test_FoodCourt_lower_bound, test_FoodCourt_upper_bound)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d2f0643-7b92-4454-a77a-c173b6a1cd87",
   "metadata": {},
   "source": [
    "#### 检查 ShoppingMall 属性是否有异常值"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe9ed5c6-e0b4-4f74-b647-868fe1fae9bf",
   "metadata": {},
   "source": [
    "1. 检查训练集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "d49b4534-df56-4708-9a1f-be1e6dc714e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_ShoppingMall_Q1 = train_data['ShoppingMall'].quantile(0.25)\n",
    "train_ShoppingMall_Q3 = train_data['ShoppingMall'].quantile(0.75)\n",
    "train_ShoppingMall_IQR = train_ShoppingMall_Q1 - train_ShoppingMall_Q3\n",
    "train_ShoppingMall_lower_bound = train_ShoppingMall_Q1 - 1.5 * train_ShoppingMall_IQR\n",
    "train_ShoppingMall_upper_bound = train_ShoppingMall_Q3 + 1.5 * train_ShoppingMall_IQR\n",
    "# 将检测到的异常值替换为边界值\n",
    "train_data['ShoppingMall'] = np.clip(train_data['ShoppingMall'], train_ShoppingMall_lower_bound, train_ShoppingMall_upper_bound)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f4051b0-1b6d-4ab8-8119-dc6a2ff7133f",
   "metadata": {},
   "source": [
    "2. 检查测试集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "cf4865cc-e047-4fe3-857f-d1e6a57e5953",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_ShoppingMall_Q1 = test_data['ShoppingMall'].quantile(0.25)\n",
    "test_ShoppingMall_Q3 = test_data['ShoppingMall'].quantile(0.75)\n",
    "test_ShoppingMall_IQR = test_ShoppingMall_Q1 - test_ShoppingMall_Q3\n",
    "test_ShoppingMall_lower_bound = test_ShoppingMall_Q1 - 1.5 * test_ShoppingMall_IQR\n",
    "test_ShoppingMall_upper_bound = test_ShoppingMall_Q3 + 1.5 * test_ShoppingMall_IQR\n",
    "# 将检测到的异常值替换为边界值\n",
    "test_data['ShoppingMall'] = np.clip(test_data['ShoppingMall'], test_ShoppingMall_lower_bound, test_ShoppingMall_upper_bound)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bde34263-ae1d-4cd1-912b-b48f0c234df7",
   "metadata": {},
   "source": [
    "#### 检查 Spa 属性是否有异常值"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a1bf2a5-8d09-479e-b5bc-4bccb085e3d2",
   "metadata": {},
   "source": [
    "1. 检查训练集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "0805ee3b-1901-4b93-9983-23a211645d04",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_Spa_Q1 = train_data['Spa'].quantile(0.25)\n",
    "train_Spa_Q3 = train_data['Spa'].quantile(0.75)\n",
    "train_Spa_IQR = train_Spa_Q1 - train_Spa_Q3\n",
    "train_Spa_lower_bound = train_Spa_Q1 - 1.5 * train_Spa_IQR\n",
    "train_Spa_upper_bound = train_Spa_Q3 + 1.5 * train_Spa_IQR\n",
    "# 将检测到的异常值替换为边界值\n",
    "train_data['Spa'] = np.clip(train_data['Spa'], train_Spa_lower_bound, train_Spa_upper_bound)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d91a6a8e-1777-4ea8-93a5-505b182c49d1",
   "metadata": {},
   "source": [
    "2. 检查测试集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "c713e4af-da2b-4b50-8386-facba15e104a",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_Spa_Q1 = test_data['Spa'].quantile(0.25)\n",
    "test_Spa_Q3 = test_data['Spa'].quantile(0.75)\n",
    "test_Spa_IQR = test_Spa_Q1 - test_Spa_Q3\n",
    "test_Spa_lower_bound = test_Spa_Q1 - 1.5 * test_Spa_IQR\n",
    "test_Spa_upper_bound = test_Spa_Q3 + 1.5 * test_Spa_IQR\n",
    "# 将检测到的异常值替换为边界值\n",
    "test_data['Spa'] = np.clip(test_data['Spa'], test_Spa_lower_bound, test_Spa_upper_bound)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cee052a-6f5b-4b4f-ac22-bdc49dfc4f18",
   "metadata": {},
   "source": [
    "#### 检查 VRDeck 属性是否有异常值"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f7b4170-77e3-419a-b3bd-de5158f2f625",
   "metadata": {},
   "source": [
    "1. 检查训练集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "d0d336e5-f44f-4dd8-bd46-7e75f31c8803",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 检查 VRDeck 属性是否有异常值\n",
    "train_VRDeck_Q1 = train_data['VRDeck'].quantile(0.25)\n",
    "train_VRDeck_Q3 = train_data['VRDeck'].quantile(0.75)\n",
    "train_VRDeck_IQR = train_VRDeck_Q1 - train_VRDeck_Q3\n",
    "train_VRDeck_lower_bound = train_VRDeck_Q1 - 1.5 * train_VRDeck_IQR\n",
    "train_VRDeck_upper_bound = train_VRDeck_Q3 + 1.5 * train_VRDeck_IQR\n",
    "# 将检测到的异常值替换为边界值\n",
    "train_data['VRDeck'] = np.clip(train_data['VRDeck'], train_VRDeck_lower_bound, train_VRDeck_upper_bound)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46a17df1-6210-4cce-bb50-d9d7f05e7e86",
   "metadata": {},
   "source": [
    "2. 检查测试集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "335d2dd2-85ec-4665-b812-2b059e03bddf",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_VRDeck_Q1 = test_data['VRDeck'].quantile(0.25)\n",
    "test_VRDeck_Q3 = test_data['VRDeck'].quantile(0.75)\n",
    "test_VRDeck_IQR = test_VRDeck_Q1 - test_VRDeck_Q3\n",
    "test_VRDeck_lower_bound = test_VRDeck_Q1 - 1.5 * test_VRDeck_IQR\n",
    "test_VRDeck_upper_bound = test_VRDeck_Q3 + 1.5 * test_VRDeck_IQR\n",
    "# 将检测到的异常值替换为边界值\n",
    "test_data['VRDeck'] = np.clip(test_data['VRDeck'], test_VRDeck_lower_bound, test_VRDeck_upper_bound)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7e357b0-d4d8-426e-b146-a1b5cc792699",
   "metadata": {},
   "source": [
    "### 5.将处理后的 data frame 输出为csv文件\n",
    "1. 将bool值转化为int类型\n",
    "2. 对非bool值标准化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "14a49e6a-343d-4d19-b366-ae5f1d13d532",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['CryoSleep', 'Age', 'VIP', 'RoomService', 'FoodCourt', 'ShoppingMall',\n",
      "       'Spa', 'VRDeck', 'HomePlanet_Earth', 'HomePlanet_Europa',\n",
      "       'HomePlanet_Mars', 'Destination_55 Cancri e',\n",
      "       'Destination_PSO J318.5-22', 'Destination_TRAPPIST-1e', 'Deck', 'Room',\n",
      "       'Side', 'Transported'],\n",
      "      dtype='object')\n",
      "Index(['CryoSleep', 'Age', 'VIP', 'RoomService', 'FoodCourt', 'ShoppingMall',\n",
      "       'Spa', 'VRDeck', 'HomePlanet_Earth', 'HomePlanet_Europa',\n",
      "       'HomePlanet_Mars', 'Destination_55 Cancri e',\n",
      "       'Destination_PSO J318.5-22', 'Destination_TRAPPIST-1e', 'Deck', 'Room',\n",
      "       'Side'],\n",
      "      dtype='object')\n"
     ]
    }
   ],
   "source": [
    "# 确保 train_data 中的标签在最后一列\n",
    "col_name = 'Transported'  # 替换为你需要移动的列名\n",
    "last_column = train_data.pop(col_name)\n",
    "train_data[col_name] = last_column\n",
    "\n",
    "# 将所有 bool类型的属性转化为 int 类型\n",
    "train_bool_columns = train_data.select_dtypes(include='bool').columns\n",
    "test_bool_columns = test_data.select_dtypes(include='bool').columns\n",
    "train_data[train_bool_columns] = train_data[train_bool_columns].astype(int)\n",
    "test_data[test_bool_columns] = test_data[test_bool_columns].astype(int)\n",
    "\n",
    "# 找出非 bool 值列\n",
    "train_non_bool_columns = train_data.columns.difference(train_bool_columns)\n",
    "test_non_bool_columns = test_data.columns.difference(test_bool_columns)\n",
    "# 对非bool值标准化\n",
    "train_scaler = StandardScaler()\n",
    "test_scaler = StandardScaler()\n",
    "train_data[train_non_bool_columns] = train_scaler.fit_transform(train_data[train_non_bool_columns])\n",
    "test_data[test_non_bool_columns] = test_scaler.fit_transform(test_data[test_non_bool_columns])\n",
    "\n",
    "print(train_data.columns)\n",
    "print(test_data.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "e42d7429-5a20-4f9a-af29-e6187f825440",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data.to_csv('RefreshedData/refreshed_train.csv', index=False)\n",
    "test_data.to_csv('RefreshedData/refreshed_test.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c4af13e-acae-49d0-a2d4-70b6a8164d37",
   "metadata": {},
   "source": [
    "### 6.结果\n",
    "经过缺省值补全和异常值处理的数据集被记录到RefreshedData目录下"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
